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A New M odel for School Accountability: 
The Urban District Asse ssment Consortium 



Russell W. Jones 
Boston College 

This document contains a brief overview and introduction to the Urban District 
Assessment Consortium (UDAC), sample, performance items and sample student 
responses, and sample open-ended items and sample student responses. Additional 
information may be obtained by contacting either UDAC, or the Center for the Study of 
Testing, Evaluation, and Educational Policy at Boston College. 

Overview 

• The Urban District Assessment Consortium (UDAC) is a multi-year cooperative effort 
funded by grants from the John D. and Catherine T. MacArthur, Ford and Boston 
Foundations and Pew Charitable Trusts. The consortium is comprised of 1 1 major 
urban school districts across the United States and the Center for the Study of Testing, 
Evaluation, and Educational Policy at Boston College. 

• Established two and a half years ago the goal of UDAC has been to develop and 
disseminate authentic assessments as alternatives to traditional standardized tests for 
school accountability purposes. While its primary purpose is to serve school level 
accountability needs, UDAC is based on the belief that the relationship between 
classroom instruction and accountability assessment should be an integrative one. 
UDAC's instruments, which include open-ended and performance tests in reading, 
writing, math and science for grades 4 and 8, have been developed to reflect emerging 
national standards in each of these four subject areas. 
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While the need for new ways of assessing student learning has been well documented, 
the vast majority of research ccnducted in this area has been limited to classroom 
assessments, such as student portfolios. There remains a dearth of knowledge 
regarding the use of these new assessments as accountability instruments. 

Much of UD AC's research and development has focused on addressing the needs of 
the "at risk" urban student population. Previous reform efforts aimed at bolstering the 
performance of these students have met wiih mixed results, and past reform measures 
focusing on assessment, such as the minimum competency and the high stakes testing 
movements, have been shown to be particularly damaging to this population of students 
(Lomax, West, Harmon, Viator & Madaus, 1992; Madaus, 1991; Shepard & 
Dougherty, 1991). With a commitment to improving the educational opportunities of 
urban students, and a firm belief in the power of expectations to influence performance 
outcomes, UDAC assessments articulate and demand rigorous performance standards 
for all students. In addition, UDAC assessments are racially, ethnically and 
linguistically inclusive, and are offered in English and Spanish to accommodate the 
language needs of the greatest number of urban students. 



The CSTEEP - School Community Partnership 

Staff at the Center for the Study of Testing, Evaluation, and Educational Policy (CSTEEP) 
at Boston College assist schools in: 

1 . Identifying educational goals and assessment strategies. 

2 . Designing and conducting training for school and district staff in the: 

i. Use of assessment techniques. 

ii. Administration of assessment instruments. 

iii. Reporting of results to parents and the community. 

3 . Designing matrix sampling techniques for school-level accountability purposes. 

Participating school districts: 

1 . Identify individuals to act as a liaison between the schools, the district and UDAC 
staff. 

2 . Accommodate the increased in-service demands of the participating schools. 

3 . Explore, in conjunction with UDAC staff, how to compare new assessment 
practices with current testing requirements. 

4 . Involve parents and the business community in school improvement efforts. 

5 . Will continue to implement the new assessment systems after the project is 
completed. 



UP AC Instruments 

The UDAC assessment administration reported here assessed the performance of 302 
fourth grade students within six Boston Public Schools, and 209 eighth grade students within 
four Boston Public Schools. Performance was assessed using performance items, long and 
shortrespon.se open-ended items, and NAEP (predominantly multiple-choice) items. This 
approach, using multiple item formats, provided teachers, students and other members of 
the school community with a wealth of information. In particular, it is worthwhile to 
highlight the large amount of information obtained from the performance items. Sample 
performance items and student responses for reading, writing, math and science 
assessments are provided in Appendix I. Sample open-ended items and student responses 
for reading, writing, math and science assessments are provided in Appendix II. 

Teachers, especially, were impressed with the quantity of pedagogical information 
obtained whilst (1) observing students perform the assessment tasks and (2) when scoring 
student responses. Because of the close ties between UDAC and parents, business and 
other community members, the broader school community also benefited from the 
assessment administrations by getting a clear indication of student abilities and performance 
across a range of tasks. 

Systematic matrix sampling was used during the assessment administration. This 
procedure permits the use of a wider range of assessment techniques than if all students were 
tested with the same asscssnic it instrument. Similar judicious use of samples has long been 
recognized in accounting, business and industry when auditing or testing for quality control. 
For example, General Motors, Toyota or Ford only crash test a sample of the cars they build. 
Yet, ihcse companies use the results of these few tests to draw conclusions concerning the 
performance of all cars they produce. Systematic matrix sampling was found to be extremely 
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effective in assessing student performance across the four subject areas (math, science, 
reading and writing) using a variety of assessment techniques. 

To meet the needs of both English and Spanish speaking students within the target 
schools, items were developed in both English and Spanish. This bilingual approach to item 
development was an important aspect of UDAC assessments. UDAC did not merely translate 
items from English to Spanish. Instead items were developed simultaneously in the two 
languages. This permitted item writers to focus on content and vocabulary from the outset 
and to immediately address any differences in syntax, semantics or concepts. Using this 
approach items were not centered in a single language or culture. Special attention was paid 
to using words and idioms that spanned the several Latino cultures, rather than focusing on 
"textbook" Spanish or a single Latino culture. Item development teams were composed of 
native speakers of both languages. These teams brought to the item development process 
diverse perspectives extremely useful for the creation of truly multicultural assessment 
instruments. The approach adopted by UDAC minimized many of the linguistic problems 
encountered by many testing programs whereby all items are developed in a single language, 
and then translated into what is usually tenned "the second language." 

Scoring Procedures 

Student responses to NAEP items were scored using NAEP answer keys. Student 
responses to performance and open-ended items were scored using a five point scoring 
rubric. A generalized rubric (shown on page 6) was developed to serve as an overall 
guide, although an item specific rubric was also developed for each item. 



Generalize d Rubric used in Raring IIP AC. 
Open-Ended and P erformance Ttems 

4 - Excellent Response: Response is correct, complete and communicated effectively. 

3 - Adequate Response: Response is essentially correct or clear despite minor errors 
or omissions. 

2 - Inadequate Response: Response is poor, incorrect, or reflects a misconception or 
miscue. 

1 - Too Brief: Too brief to evaluate, no response, item left blank, restates question, "I 
don't know." 

0 - Student Absent. 

This rubric applies uniformly across subject areas and grade levels. Sample answers are 
provided in the item specific scoring keys. 



Rubrics were developed under the guidance of subject matter specialists. Item 
writing and rubric development occurred concurrently. Scoring was performed by 
teachers, parents, other community members, and UDAC staff. Prior to the scoring of 
student responses, training sessions were held to train personnel in scoring procedures and 
to provide practice in rating student responses. 

Rater Teams 

Items were scored by three member teams comprised of a UDAC representative, a 
school representative (usually a teacher or administrator), and a member selected from the 
community. Each performance and open-ended item was independently scored by two 
members of a rating team. Following this initial independent scoring, team members met 
and compared their scores. If discrepancies in scoring between two raters occurred, the 
two niters discussed their reasons for assigning a particular score in an attempt to reach 
consensus. Typically this discussion resulted in raters agreeing to alter one or both ratines 
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to reach consensus. Consensus was reached when the two raters agreed on a score. If 
consensus was not reached both scores were recorded. 



Summary of Results 

Analysis of UDAC performance and long and short response open-ended items 
showed most functioned very well. Item difficulty values for performance items covered a 
broad range from .33 to .69 in reading, from 0 to .69 in science, from 0 to 1.0 in math and 
from .26 to .47 in writing. Item difficulty values for open-ended items also covered a broad 
range from 0 to .73 in reading, from 0 to .87 in science, from 0 to .88 in math and from 0 to 
.83 in writing. Thus, for each subject, the assessment instruments were able to assess students 
with a wide range of ability levelsfrom very weak to very strong. Item discrimination values 
for most items were generally well above the widely accepted value of 0.3. Indeed, the mean 
item discrimination value for UDAC performance items was .50 in reading, .35 in science, .47 
in math, and .37 in writing. However, values for open-ended items were generally lower; .46 
in. reading, .27 in science, .23 in math, and .38 in writing. Thus, UDAC items provided a 
great deal of information and were able to differentiate between students having various levels 
of ability in each of the subject areas. Statistical analysis of item properties identified a small 
proportion of items that appeared to be functioning poorly. These items have been revised or 
eliminated prior to further test administrations. 

In addition to the ability of UDAC items to assess and discriminate well between 
students of varying ability, an evaluation of the relationship between the Metropolitan 
Achievement Test (MET) (a standardized test given to students in the Boston Public Schools 
district) and NAEP standardized tests shows UDAC items provide different infonnation to that 
provided by these standardized tests. All three assessments were administered to students 
(luring Hie same semester. The NAEP and MET assessments arc both established testing tools 
widely accepted as suitable measurement tools to evaluate student ability (for example, NAEP 



tests rank among the most established and broadly applied tests in the United States having 
been used to evaluate the national performance of American students for more than 20 
years). Correlational analyses between these two assessments and UDAC assessments, revealed 
a moderate (for example, r = .31) to high (for example, r = .66) positive relationship between 
student performance on the MET and NAEP tests and their performance on the UDAC 
assessments. This relationship provides powerful criterion-related validity for the newer 
UDAC assessments. However, the finding that this relationship is not perfect clearly indicates 
UDAC assessments provide a substantial amount of information different to that provided by 
either the MET or NAEP standardized tests. 

School faculty and community members were veiy appreciative of the wealth of 
information provided by UDAC items. By participating in the administration and scoring of 
student responses they were presented with clear demonstrations of student ability. In other 
words, educational feedback was not limited to post analysis consideration of assessment 
results, instead the entire assessment process was a learning activity for all individuals who 
were associated with the assessments. Evaluation of the UDAC program included written 
comments by school faculty. These comments point to the benefits obtained as a result of the 
UDAC assessments. In the words of one principal "The greatest asset (of these assessments) is 
that there are implications for teaching!" 
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APPENDIX I 
Sample Performance Items and Student Responses 
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APPENDIX II 
Sample Open-Ended Items and Student Responses 
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1. Tomas' teacher ordered lunch for his class from a local sandwich shop. She ordered the lunches 
from the menu below. 

Sandwiches Milkshakes 



ham & cheese chocolate 

roast beef vanilla 

turkey club strawberry 

bacon, lettuce & tomato pineapple 




If a lunch consists of one sandwich and one milkshake, how many different lunch 
combinations are possible? [EMM30A2-22] 



Answer: 



it, 



2. If the teacher ordered one of each of the possible lunch combinations, what is the probability that 
Tomas would receive a turkey club sandwich and a chocolate milkshake for lunch? [EMM30A2-23] 



Answer: 
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4. Below is a third grader's addition homework. Look carefully at each answer. 
The student is very confused, 

/ 

29 44 

+_3 +_6 

212 410 



\ 



18 67 
+ 67 +84 
715 1411 



On the lines below, write down everything you think this student did wrong and 
how you would help the student understand why the answers are wrong. 



J lAS4- jp-iA/l: ^cRi_ h/A^v\Uv a/' IrL,. h>j> C/Visfet J 

Ug_ V K; / I > 



[EPM30A2-25] 
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Directions: Read each question carefully before answering. 



Jane poured some water 
into the U-shaped glass tube as 
shown here. However, there was 
not enough water to completely 
fill the tube. 



<2b 



v.uj,-,.j,i.mj.mB 



3 - Draw on the picture below what the water level in the tube will look like when all the water has 
been poured in. [EMS25A2-3] 





Why do you think it will look the way you have drawn it? [EMS30A2-4] 

c\fW^/ Pall* Wck^I Jou/n ,V w 0 \)\ds\V 
^<av^ Sftf/t^cl on -lit loq<\ r JJlg_ of 
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Directions: Think about each question carefully before writing your answer. 



Kim placed a stick in the ground on 
a sunny day. Then she measured the 
length of the stick's shadow at four 
different times. She made a graph of 
the measurements, which is shown 
to the right. 




8un 9im 10 am Ham 12 noon 



J 



About how long was the shadow of the stick at M a.m.? Circle the correct answer below. 



A. 

C. 
D. 



10 cm. 
20 cm. 
30 cm. 
40 cm. 




6. Why did the shadow get shorter and shorter? 

tL 



[EMS20A2-5] 
(EMS30A2-6] 




J 
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READING MIDDIlE LEVEL SbOOKPA2 



Directions: The following passage is taken from a book by Rigoberta Menchu. Read it 
carefully and answer the questions that follow. 



Rigoberta Menchu is a woman from Guatemala. She wrote the book called L Rigoberta Menchu 
An Indian Woman In Guatemala . In 1992, she won the Nobel Peace Prize. She received this award 
because she spent most of her life fighting the terrible injustices she and her people suffered. In the 
following paragraphs, she tells us a little about her life as a child and the kind of work she and her 
mother did in Guatemala. 



special vocabulary: finca: a farm 

tortillas: unleavened bread made from cornmeal 
nixtamal: dough used to make tortillas 



"An Eight- Year-Old Agricultural Worker" 

'And that's when my consciousness was born 

-Rigoberta Menchu 

I worked from when I was very small, but I didn't earn anything. I was really helping my 
mother because she always had to cany a baby, my little brother, on her back as she picked coffee. 
It made me very sad to see my mother's face covered in sweat as she tried to finish her work load, 
and I wanted to help her. But my work wasn't paid; it just contributed to my mother's work. I 
either picked coffee with her or looked after my little brother, so she could work faster. My 
brother was two at the time.... 

I remember that, sometimes, my mother's work was making food for forty workers. She 
ground maize, made tortillas, put the nixtamal on the fire and cooked beans for the workers' food, j 
That's a difficult job in [he finca. All the dough made in the morning has to be finished the same 
morning because it goes bad. My mother had to make the number of tortillas the workers would j 
eat. She was very appreciated by the workers because the food she gave them was fresh. The food 
we ate was cooked by another woman who sometimes gave us things that had gone bad, or torti- 
llas which were tough and beans which jumped when you tried to pick them up. In the finca the 
women who do the cooking don't know which people they will cook for. The overseer comes and \ 
says, This is your group ... this is what you give them to eat, these are the people you feed, you 
feed them at such and such a time ... so get to work.' So different women fed us. My mother liked 
to give the workers the food ihcy deserved, even if it meant she didn't sleep all night. They came 
back tired from the fields and she wanted to see that they ate well, even though her own family 
was eating badly somewhere else. 

I was five when she was doing this work and I looked after my little brother. I wasn't 

■» """" " ^ ^ — — I 



GO ON TO THE NEXT PAGE 



9 

ERJC 



34 



READING 



MIDDLE LEVEL 



BOOKA2 



earning yet. I used to watch my mother, who often had the food ready at five o'clock in the 
morning for the workers who started work early, and at eleven she had the food for the midday 
meal ready. At seven in the evening she had to run around again making food for her group. In 
between times, she worked picking coffee to supplement what she earned. Watching her made 
me feel useless and weak because I couldn't do anything to help her except look after my 
brother. That's when my conciousness was born. It's true. My mother didn't like the idea of me 
working, but I did. I wanted to work, more than anything to help her, both economically and 
physically.... 



What kind of work did Rigoberta do when she was a child? 



[EMR20A2-14] 

What is an agricultural worker? 

0\ plAfln -/-W W0 r ^^ WiA \wJ PC jffl/Vw / 



[EMR25A2-15] 

Why was her mother's job so difficult? 



ft 



Why was it so important for Rigoberta to help her mother? 

k UA l-rr work ■ - : 



IEMR25A2-16] 



|i-:.mk25a: ;:; 
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Directions: Read the following questions and write your answers in the space provided. 



When people buy something and it breaks or doesn't do what it is supposed to do, they often 
write a letter complaining to the manufacturer about the product Pretend you bought a very 
expensive pair of sneakers that fell apart the second day you wore them. On the lines below, 
write a letter to the company that makes the sneakers. Let the company know: 

* what kind of sneakers you bought 

* what was wrong with them, and 

* what you want the company to do about it. 

Make sure your letter is a business letter that includes the name and address of the sneaker 
company, a date, a greeting, the body of the letter, a closing and a signature. If you need more 
space, use the next page. 




X dm£ fx bcAt&n 



-2^ (a ^ r ■' ■; 6 g( M # £ 5 o La C 7. rnf (\ Lfjg f •/ 
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X would I.' kg. g rt-fanj fl/)cA <k 
po^if of &mj;*y .ffj»6tf, 




2. 



[EMWPA2-9] 

In a paragraph, explain why you think the letter you wrote will make the company respond to 
your complaint. 



IKMWIA2 HJj 
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